breaks, likelihood of the presence of a commercial at any 

given time in a set of audiovisual content). Optimization 

can be done over the total set of commercial location 

i 

decisions, rather than on a per-commercial basis. 
Additionally, the cost function can be iteratively 
evaluated, increasing the accuracy of commercial location 
decisions produced by the method. Additionally, many more 
types of cues and combinations of cues can be used in 
detection of commercials in accordance with the invention 
than have been used in other approaches to commercial 
detection . 



REMARKS 

The specification and abstract have been amended to correct 
grammatical and idiomatic errors. Applicant requests entry of 
the amendments to the specification and abstract. 

Claims 1 2-68 have been added to more fully claim the 
invention. Support for Claims 12-15 is found in Applicant's 
specification at, for example, page 14, line 28 to page 16, 
line 21. Support for Claims 16-20 is found in Applicant ' s 
specification at, for example, page 16, line 22 to page 18, 
line 30. Support for Claims 21-25 is found in Applicant's 
specification at, for example, page 18, line 31 to page 19, 
line 23. Support for Claim 26 is found in Applicant's 
specification at, for example, page 18, lines 20-25. Support for 
Claim 27 is found in 'Applicant's specification at, for example, 
page 21, lines 23-24. Support for Claims 28-32 is found in 
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Applicant's specification at, for example, page 21, line 25 to 
page 22, line 8. Support for Claim 33 is found in Applicant's 
specification at, for example, page 22, line 9 to page 23, 
line 19. Support for Claim 34 is found in Applicant's 
specification at, for example, page 23, line 20 to page 25, 
line 18. Support for Claims 35-39 is found in Applicant's 
specification at, for example, page 5, lines 4-12 and page 8, 
line 30 to page 9, lijne 6. Support for Claims 40-43 is found in 
Applicant's specification at, for example, page 26, lines 27-33. 
Support for Claim 44 is found in Applicant's specification at, 
for example, page 26, line 33 to page 27, line 2 and page 25, 
line 19 to page 26, line 20. Support for Claims 45-46 is found 
in Applicant's specification at, for example, page 4, lines 12-14 
and page 6, line 16 to page 8, line 8, as well as generally 
throughout Applicant's specification. Support for Claims 47-49 
is found in Applicant's specification at, for example, page 4, 
lines 14-20, as well as generally throughout Applicant's 
specification. Support for Claims 50-51 is found in Applicant's 
specification at, fo^ example, page 4, lines 22-31 , as well as 
generally throughout Applicant's specification. Support for 
Claims 52-68 is found in Applicant's specification at, for 
example, page 14, lines 9-27. 

Claims 1-11 were pending. Claims 1 and 8-11 have been 
amended. Claims 12-68 have been added. Allowance of Claims 1-68 
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is requested. If the Examiner wishes to discuss any aspect of 
this application, the Examiner is invited to telephone 
Applicant's undersigned attorney at (408) 945-9912. 



I hereby certify that this correspondence is being Respectfully Submi 
deposited with the United States Postal Service as 

first class mail in an envelope addressed to: CL < a j1 

Assistant Commissioner for Patents, Washington, Z J A M II P WiAi 



D.C. 20231, on November 30, 2001 . 





David R. Graham 
Reg. No. 36,150 
Attorney for Applicant 
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Version with Markings to Show Changes Made 
(Additions are underlined, deletions are enclosed in brackets) 



In the specification: 

The paragraph beginning at page 7, line 4 has been amended 
as follows: 

In step 103 of the method 100, the scores associated with 
each candidate time are adjusted. The score for a candidate time 
can be adjusted, for example, based on an analysis of one or more 
cues proximate to the candidate time that are different [than] 
from the one or more cues used to identify the candidate time. 
For instance, as described in more detail below with respect to 
step 206 of the method 200 (see Figure 2) the score associated 
with a candidate time can be adjusted based on the presence or 
absence of one or more cues within. a specified time window that 
includes the candidate time or to which the candidate time is 
sufficiently proximate (i.e., is less than a specified short 
amount of time, such as several seconds, before or after the time 
window). The score for a candidate time can also be adjusted, 
for example, based on an evaluation of the relationship between 
the candidate time and one or more other candidate times. In 
particular, as described in more detail below with respect to 
step 207 of the methoid 200 (see Figure 2), this latter type of 
adjustment can make use of one or more probability models that 
describe expected relationship^ ) between a candidate time and 
the one or more other candidate times. 
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The paragraph beginning at page 11, line 16 has been amended 
as follows: 

In step 205 of the method 200, one or more of the cues 

identified in step 204 are analyzed to identify candidate times 

within the audiovisual content at which a commercial beginning or 

i 

a commercial ending may occur. For example, an audio pause often 
accompanies either the beginning or the end of a commercial, so 
the presence of an audio pause in the audio content can be 
identified as a factor that militates toward establishing a 
candidate time at some time during or proximate to the audio 
pause. Similarly, a sequence of black frames often accompanies 
either the beginning or the end of a commercial, so the presence 
of a sequence of black frames in the visual content can be 
identified as a factor that militates toward establishing a 
candidate time at some time during or proximate to the sequence 
of black frames. A scene cut or fade also typically accompanies 
the beginning or the end of a commercial, so the presence of a 
scene break or fade in the visual content can be identified as a 
factor that militates toward establishing a candidate time at 
some time during or proximate to the scene break or fade. The 
beginning and end of a commercial break are often accompanied by 
a noticeable increase and decrease in volume, respectively, so 
that a significant change in average volume (measured over a 
specified window of time) can be identified as a factor that 
militates toward establishing a candidate time at some time 
proximate to times at which the volume is seen to change 
significantly. Commercials often include relatively more musical 



1 

content than the rest of a set of audiovisual content, so the 
occurrence of a time window of specified duration (e.g., the 
expected duration of a typical commercial break, such as 60 
seconds, or the expected duration of a typical commercial, such 
as 1 5 or 30 seconds) having relatively high musical content 
(e.g., relatively high density of musical content relative to the 
density of musical content in other, proximate time windows) can 
be identified as a factor that militates toward establishing 
candidate times at the beginning and end of such a time window. 
The beginning or end pf a commercial is often accompanied by a 
change in speaker identity, so the occurrence of a change in 
speaker identity can be identified as a factor that militates 
toward establishing a candidate time at, or proximate to, the 
time at which such a change in speaker identity occurs. A 
commercial break often includes a relatively high density of 
scene breaks and/or fades (since a scene, break or fade typically 
occurs at the beginning and end of a commercial break, as well as 
at the transition between commercials within a commercial break, 
and since commercials often include a relatively large number of 
scene breaks and/or fades per unit time within the commercial), 
so the occurrence of a time window of a specified duration 
(e.g., 60 seconds) during which the density of scene breaks 
and/or scene fades is relatively high (i.e., exceeds a specified 
threshold), or a significant change in density of scene breaks 
and/or scene fades over one window of time with respect to a 
proximate window of time, can be identified as a factor that 
militates toward establishing candidate times at the beginning 
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and end of such a time window. A network icon is sometimes 
present during the noncommercial parts of a television broadcast; 
therefore, if a network icon is determined to be present in a set 
of audiovisual content, the disappearance of the network icon 
typically accompanies the beginning of a commercial break and the 
appearance of the network icon typically accompanies the end of a 
commercial break, so the appearance or disappearance of a network 
icon can be identified as a factor that militates toward 
establishing a candidate time at, or proximate to, a time at 
which the network icon appears or disappears. Since the average 
motion level in the visual content of a commercial is often 
significantly different [than] from the average motion level of 
other visual content in a set of audiovisual content, significant 
change in the amount of motion in the visual content of a time 
window (e.g., about 6'0 seconds) relative to the amount of motion 
in the visual content in a proximate time window can be 
identified as a factor that militates toward establishing 
candidate times at, or proximate to, the beginning and end of 
such a time window. The appearance of text (other than 
closed-captioning) in a set of audiovisual content often 
accompanies the beginning of a commercial break and the 
disappearance of text often accompanies the end of a commercial 
break, so the appearance or disappearance in a set of audiovisual 
content of text other than closed-captioning can be identified as 
a factor that militates toward establishing a candidate time at, 
or proximate to, a time at which text appears or disappears. If 
closed-captioning data is present in the data representing the 



audiovisual content, a closed-captioning scrolling format change 

often occurs at the beginning or the end of a commercial break, 

so the occurrence of a closed-captioning scrolling format change 

can be identified as a factor that militates toward establishing 

a candidate time at, or proximate to, the time at which such a 

format change occurs. If closed-captioning data is present in 

the data representing the audiovisual content, the disappearance 

of closed-captioning often accompanies the beginning of a 

i 

commercial break and the appearance of closed-captioning often 
accompanies the end of a commercial break, so the appearance or 
disappearance of closed-captioning can be identified as a factor 
that militates toward establishing a candidate time at, or 
proximate to, a time at which closed-captioning appears or 
disappears . 

The paragraph beginning at page 14, line 9 has been amended 
as follows: 

As indicated above, it is an advantageous aspect of the 

invention that the invention enables use of a combination of the 

i 

cues to effect commercial detection. In particular, the 
invention can enable the use of cues and combinations of cues 
that have not previously been used for commercial detection. For 
example, the invention can advantageously enable any one of 
detection of the absence of a network icon, an analysis of 
musical content present in a set of audiovisual content, the 
density of scene breaks and/or fades, or analysis of the identity 
of speakers of spoken content to be used alone as a commercial 
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detection cue. These cues can also be used in any combination 
with each other or anty other cue. In particular, it is 
anticipated that one or more of these cues can advantageously be 
used in combination with one or more of the following 
cues: 1) the occurrence of an audio pause, 2) the occurrence 
of a sequence of black frames, 3) a scene cut or fade, 4) the 
occurrence of specified closed-captioning formatting signals, 
and 5) the appearance or disappearance of closed-captioning. 

The paragraph beginning at page 14, line 28 has been amended 
as follows: 

Step 205 outputs a list of candidate times at which 
commercials may be beginning or ending, together with a score or 
probability associated with each candidate time. In one 
implementation of the invention, each candidate time is assigned 
the same initial score. Alternatively, the scores assigned to 
candidate times can vary. For example, the score for a candidate 
time can depend on which cue(s) were used to identify the 
candidate time. The beginning or end of a commercial can be 
deduced from the presence of some cues with a greater degree of 
confidence than that associated with the presence of other cues. 
To the extent that a candidate time is identified based on a cue 
with which a relatively high degree of predictive confidence is 
associated, the scorej assigned to that candidate time can be 
relatively higher than would be the case if a relatively low 
degree of predictive confidence was associated with the cue. 
Additionally, the score for each candidate time can be dependent 
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on how strongly the cue is present in the audiovisual content, as 
determined in accordance with a criterion or criteria appropriate 
for that cue: the more strongly a cue is present, the higher the 
score. For example, when one of the cues used to establish a 
candidate time is an audio pause, the score established for the 
candidate time can be dependent on the duration of the audio 
pause and/or the degree of silence during the audio pause (e.g., 
the score for the candidate time is made relatively greater the 
longer the audio pause or the less sound that is present during 
the audio pause). Or, for example, when one of the cues used to 
establish a candidate time is a sequence of black frames, the 
score established for the candidate time can be dependent on the 
duration of the sequence of black frames and/or the completeness 
of the blackness of the frames (e.g., the score for the candidate 
time is made relatively greater the longer or blacker the 
sequence of black frames). Or, for example, when one of the cues 
used to establish a candidate time is a scene cut, the score 
established for the candidate time can be dependent on the number 
of pixels that changed by more than a threshold amount from one 
frame to another (e.g., the score for the candidate time is made 
relatively greater as more pixels changed between scenes) and/or 
dependent on the total change of all the pixels from one frame to 
another (where the "change" for each pixel is the change in the 
color or other components of a pixel). Or, for example, when one 
of the cues used to establish a candidate time is a significant 
average audio volume change, the score established for the 
candidate time can be dependent on degree of the volume change 
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(e.g., the score for the candidate time is made relatively 
greater as degree of the volume change increases). Those skilled 
in the art can readily appreciate how the score for a candidate 
time can be adjusted based on aspects of other cues present in 
the audiovisual content proximate to the candidate time. 
Additionally, the score for a candidate time can be dependent on 
the confidence level associated with identification of the cue in 
the audiovisual content: the greater the confidence level, the 
higher the score. (This confidence level is different [than] 
from the confidence level associated with the predictive 
capability of the cue, discussed above.) For example, sound 
represented in audio data may be sound in the audio content or 
noise. The score for a candidate time identified at least in 
part based on the presence of an audio pause can be increased or 
decreased in accordance with extent to which the degree of noise 
present in the audio data increases or decreases the confidence 
with which an audio pause can be detected. 

The paragraph beginning at page 16, line 22 has been amended 
as follows: 

In step 206 of the method 200, the scores associated with 

each candidate time can be adjusted based on the presence or 

i 

absence of one or more cues within some time window proximate to 
the candidate time. The cue(s) used to adjust the score of a 
candidate time in step 206 are different [than] from the cue(s) 
used to establish the candidate time and an initial associated 
score in step 205. The duration of the time window and location 
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of the time window with respect to the [cue] candidate time is 
dependent on the type of cue. For instance, the score for a 
candidate time is increased (i.e., the likelihood that the 
candidate time correctly indicates the beginning or ending of a 
commercial is deemed to increase) in each of the following 
cases: 1) the candidate time is coincident with the time at 
which an audio pause (which is a window of audio silence or near 
silence) occurs, 2) the candidate time is within or sufficiently 
proximate to a time window in which the closed-captioning 
scrolling format is different from that which is typical for 
audiovisual content of this type, 3) the candidate time is within 
or sufficiently proximate to a time window during which 
closed-captioning is absent (for audiovisual content that is 
known to be closed-captioned ) , 4) the candidate time is within or 
sufficiently proximate to a time window of at least a specified 
duration (e.g., 60 seconds) and including high musical 
content, 5) the candidate time is within or sufficiently 
proximate to a time window during which the density of scene 
breaks and/or scene fades exceeds a specified threshold, 6) the 
candidate time is sufficiently proximate to a time window of at 
least a specified duration (e.g., 0.5 seconds) and in which the 
average motion in the visual content, measured in a specified 
manner, is less than a specified threshold, 7) the candidate time 
is within a time window during which a network icon (which has 
been found to be persistent through a majority of the visual 
content) is not present at a specified location within the visual 
content (e.g., a regipn, such as a corner, near the edge of the 



visual content), 8) the candidate time is very near (e.g., within 
about 2 seconds) a time at which the time-averaged audio volume 
(averaged over a time window of about 10 seconds) has changed by 
a magnitude of greater than a specified threshold, 9) the 
candidate time is sufficiently proximate to (within about 1 
second) a time when text is present in the visual 

content, 10) the candidate time is within a specified duration of 
time (e.g., a few seconds) after the presence in the 

closed-captioning stream of certain keywords or phrases such as 

i 

"commercial", "break", "coming up" or "after", or within a 
specified duration of time (e.g., a few seconds) prior to the 
presence in the closed-cap I ioning stream of certain keywords or 
phrases such as "welcome", "hello" or "we're back, 11) the 
candidate time is within a specified duration of time (e.g., 2 
seconds) from a time at which the speaker identity has changed, 
and 12) the candidate time is within a specified duration of time 
(e.g., one to several seconds) from a time window of greater than 
a specified duration (e.g., 1 minute) that does not include 
speech from a speaker whose speech has been determined to be 
present in the audiovisual content with greater than a specified 
frequency. The amount by which a score is adjusted can be 
dependent on the same types of analyses done to establish an 
initial score for a candidate time, as described above with 
respect to step 205. (However, the particular analyses done in 
step 206 need not, but can^ be[,] the same as those done in 
step 205.) In particular, the amount of the adjustment to a 
score for a candidate time can be dependent on how strongly the 



cue is present in the audiovisual content, as determined in 
accordance with a criterion or criteria appropriate for that cue 
in general, the more strongly a cue is present, the greater the 
adjustment to the score. Additionally, the amount of the 
adjustment to a score for a candidate time can be dependent on 
how high or low the score is prior to the adjustment. For 
example, a cue that strongly indicates the presence of a 
commercial beginning or ending may cause a larger adjustment in 
relatively low score than in a relatively high score. The 
particular quantities, keywords, and other algorithm parameters 
given above are illustrative; they may be changed, within 
aonronriahft constraints, as can be aonreciated bv those skilled 
in the art, without adversely affecting the operation of the 
invention. 

i 

The paragraph beginning at page 22, line 9 has been amended 
as follows: 

Step 208 begins by selecting the candidate time with the 
highest score to be a commercial start or end time (whether that 
time is a start time or end time is unknown at this point). A 
commercial break is then constructed based on the selected 
candidate time by successively evaluating candidate times in 
order of decreasing score and adding candidate times to the 
commercial break that satisfy each of the following 

criteria: 1) the additional candidate time is well-spaced in 

i 

time, in accordance with the function S(t), from each candidate 
time that has already been included in the commercial 



break, 2) the additional candidate time does not create a 
commercial break which is too long, in accordance with the 
function L(t), and 3) the additional candidate time is not too 
close to other existing commercial breaks, in accordance with the 
function W(t), that have already been identified by the step 208. 
Stated another way, candidate times continue to be added to a 
commercial break in order of score as long as there are any 
candidate times for which all of the following are true: 1) the 
value of S(t), where "t" is the time separation between the 
candidate time being evaluated and a candidate time already 
included in the commercial break, is above a specified threshold 
value for each candidate time already included in the commercial 
break, 2) the value of L(t), where "t" is the duration of the 
commercial break if the candidate time is added to the commercial 
break, is [not below] above a specified threshold value, and 3) 
the value of W(t), where "t" is the time separation between the 
candidate time and an existing commercial break, is [not below] 
above a specified threshold value for each existing commercial 
break. t 

The paragraph beginning at page 26, line 25 has been amended 
as follows: 

The invention can be used for a wide variety of 
applications, as can be appreciated by those skilled in the art 
in view of the description herein. In general, the invention can 
be used with any broadcast or other data transmission over a 
network (e.g., conventional network television broadcasts, cable 
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television broadcasts, broadcasts or transmissions over a 

i 

computer network such, as the Internet - and, in particular, the 

World Wide Web portion of the Internet). Additionally, the 

invention can be used generally to detect commercials in 

audiovisual content represented by any type of data, which data 

can be stored on a data storage medium or media, or provided to a 

system or method according to the invention in real time. 

Further, the invention can be implemented in a wide variety of 

apparatus, as can also be appreciated by those skilled in the art 

in view of the description herein, such as, for example, 

television set-top boxes, digital VCRs, computers (including 

desktop, portable or handheld computers) or any of a variety of 

i 

other computational devices (including many which are now being , 
or will in the future be, developed). 

In the claims : 

Claims 1 and 8-11 have been amended as follows: 

1 . (Amended) A method for detecting one or more commercial 

breaks in a set of audiovisual content spanning a duration of 

time , each commercial break including one or more commercials, 

the method comprising the steps of: 

identifying candidate times within the duration of time 

spanned by the set of audiovisual content based on an 

i 

evaluation of one or more cues identified in the audiovisual 
content, each candidate time representing a possible 
starting [time or a possible] and/or ending time of a 
commercial ; 
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assigning a score to each candidate time; 

evaluating, for each of one or more candidate 
times, 1) one or more secondary cues [other than] that are 
each different from the one or more cues used to identify 
the candidate time, and/or 2) the relationship between the 
candidate time atad one or more other candidate times, 
wherein the score assigned to the candidate time can be 
adjusted based on the evaluation; and 

constructing the one or more commercial breaks based on 
an evaluation of 1 ) the scores of the candidate times after 
the step of evaluating and 2) [the] a relationship between, 
or relationships among^ the candidate times. 

8. (Amended) A system for detecting one or more commercial 
breaks in a set of audiovisual content spanning a duration of 
time , each commercial break including one or more commercials, 
the system comprising^ 

[means for identifying the presence of one or more cues 
in the audiovisual content;] 

means for identifying candidate times within the 
duration of time spanned by the set of audiovisual content 
based on an evaluation of one or more cues identified in the 
audiovisual content, each candidate time representing a 
possible starting [time or a possible] and/or ending time of 
a commercial; 

means for assigning a score to each candidate time; 



- 57 - 
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means for evaluating, for each of one or more candidate 
times, 1 ) one or more secondary cues [other than] that are 
each different from the one or more cues used to identify 
the candidate time, and/or 2) the relationship between the 
candidate time and one or more other candidate times, 
wherein the score assigned to the candidate time can be 
adjusted based on the evaluation; and 

means for constructing the one or more commercial 

breaks based on an evaluation of 1 ) the scores of the 

candidate times after the step of evaluating and 2) [the] a 

t 

relationship between, or relationships among^ the candidate 
times. 

9. (Amended) A computer readable storage medium or media 
encoded with one or more computer programs including instructions 
for detecting one or more commercial breaks in a set of 
audiovisual content spanning a duration of time , each commercial 
break including one or more commercials, the one or more computer 
programs comprising : 

[instructions for identifying the presence of one or 
more cues in the audiovisual content; ] 

instruction's for identifying candidate times within the 
duration of time spanned by the set of audiovisual content 
based on an evaluation of one or more cues identified in the 
audiovisual content, each candidate time representing a 
possible starting [time or a possible] and/or ending time of 
a commercial; 
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instructions for assigning a score to each candidate 

time; 

instructions for evaluating, for each of one or more 
candidate times, ^ 1 ) one or more secondary cues [other than] 
that are each different from the one or more cues used to 
identify the candidate time, and/or 2) the relationship 
between the candidate time and one or more other candidate 
times, wherein the score assigned to the candidate time can 
be adjusted based on the evaluation; and 

instructions for constructing the one or more 
commercial breaks based on an evaluation of 1 ) the scores of 
the uariuidaLe times after the step of evaluating and 2) 
[the] a relationship between, or relationships among^ the 
candidate times. 

» 

10. (Amended) A method for detecting one or more commercial 
breaks in a set of audiovisual content spanning a duration of 
time, each commercial break including one or more commercials, 
the method comprising the steps of: 

identifying candidate times within the duration of time 
spanned by the set of audiovisual content based on an 
evaluation of one or more cues identified in the audiovisual 
content, each candidate time representing a possible 
starting [time or a possible] and /or ending time of a 
commercial ; 

assigning a score to each candidate time; and 
i 
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constructing the one or more commercial breaks based on 
an evaluation of 1 ) the scores of the candidate times after 
the step of evaluating and 2) [the] a relationship between, 
or relationships among^ the candidate times. 

11. (Amended) A method for detecting one or more commercial 
breaks in a set of audiovisual content spanning a duration of 
time , each commercial break including one or more commercials, 
wherein candidate timtes, each candidate time representing a 
possible starting [time or a possible] and/or ending time of a 
commercial within the duration of time spanned by the set of 
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of one or more cues identified in the audiovisual content and a 
score assigned to each candidate time, the method comprising the 
steps of: 

evaluating, for each of one or more candidate 
times, 1 ) one or more secondary cues [other than] that are 
each different from the one or more cues used to identify 
the candidate time, and/or 2) the relationship between the 
candidate time amd one or more other candidate times, 
wherein the scorie assigned to the candidate time can be 
adjusted based on the evaluation; and 

constructing the one or more commercial breaks based on 
an evaluation of 1 ) the scores of the candidate times after 
the step of evaluating and 2) [the] a relationship between, 
or relationships among^_ the candidate times. 
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Claims 12-68 have been added as follows: 

12. (New) A method as in Claim 1, wherein the step of 
assigning further comprises the step of assigning the same score 
to each candidate time. 

13. (New) A method as in Claim 1, wherein the step of 
assigning further comprises the step of assigning a score to each 
candidate time in accordance with the type of cue or cues 
evaluated to identify the candidate time. 

14. (New) A method as in Claim 1, wherein the step of 

t 

assiqning further comprises the step of assigning a score to each 
candidate time in accordance with the degree of presence in the 
audiovisual content of the cue or cues evaluated to identify the 
candidate time. 

15. (New) A method as in Claim 1, wherein the step of 
assigning further comprises the step of assigning a score to each 
candidate time in accordance with the degree of confidence of 
identification of the cue or cues evaluated to identify the 
candidate time. 

16. (New) A method as in Claim 1, wherein the step of 
evaluating further comprises the step of determining the presence 
or absence of a secondary cue within a time window that includes 
the candidate time or to which the' candidate time is sufficiently 
proximate . 
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17. (New) A method as in Claim 16, wherein the duration 
and/or location of the time window depends on the type of the 
secondary cue. 



18. (New) A method as in Claim 16, wherein the score is 
adjusted in accordant with the type of the secondary cue. 

19. (New) A method as in Claim 16, wherein the score is 
adjusted in accordance with the degree of presence of the 
secondary cue. 

20. (New) A method as in Claim 16, wherein the score is 
adjusted in accordance with the degree of confidence of 
identification of the secondary cue. 

21. (New) A method as in Claim 1, wherein the step of 
evaluating further comprises the step of comparing the candidate 
time and/or a relationship between, or relationships among, the 
candidate time and one or more other candidate times to one or 
more probability models that specify one or more expected 
characteristics of commercial start and/or end times, and/or an 
expected relationship between, or relationships among, commercial 
start and/or end times. 

22. (New) A method as in Claim 21, wherein one of the 
probability models specifies the expected temporal separation of 
commercial start and end times. 



23. (New) A method as in Claim 21, wherein one of the 
probability models specifies the expected location of one or more 
commercial start and/or end times within the duration of time 
spanned by the set of audiovisual content. 

24. (New) A method as in Claim 21 , wherein one of the 
probability models is derived from statistics concerning any type 
of audiovisual content. 

25. (New) A method as in Claim 21, wherein one of the 
probability models is derived from statistics concerning only 
audiovisual content that is of the same type ay the set of 
audiovisual content in which the one or more commercials breaks 
are being detected. 

26. (New) A method as in Claim 1, wherein the score 
adjustment varies in accordance with the magnitude of the score 
before adjustment . 

27. (New) A method as in Claim 1, further comprising the 
step of eliminating candidate times having an adjusted score 
below a specified threshold. 
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28. (New) A method as in Claim 1, wherein the step of 

constructing further comprises the step of comparing a 

relationship between, or relationships among, the candidate times 

i 

to one or more probability models that specify an expected 
relationship between, or relationships among, commercial start 
and/or end times. 

29. (New) A method as in Claim 28, wherein one of the 
probability models specifies the expected duration of a 
commercial break. 

30. (New) A method as in Claim 26, wherein one of the 
probability models specifies the expected temporal separation of 
commercial breaks . 

i 

31 . (New) A method as in Claim 28, wherein one of the 
probability models is derived from statistics concerning any type 
of audiovisual content. 

32. (New) A method as in Claim 28, wherein one of the 
probability models is derived from statistics concerning only 
audiovisual content that is of the same type as the set of 
audiovisual content in which the one or more commercials breaks 
are being detected. 

i 
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33. (New) A method as in Claim 28, wherein the step of 
constructing further comprises the steps of: 

selecting the candidate time with the highest adjusted 
score to be a commercial start and/or end time in a current 
commercial break; 

successively evaluating each candidate time not yet 
part of a commercial break, in order of decreasing adjusted 
score, for possible inclusion in the current commercial 
break as a commercial start and/or end time, wherein the 
evaluation of each candidate time for possible inclusion in 
the current commercial break comprises the steps of: 
determining whether each probability of the 
temporal separation between the candidate time being 
evaluated and a candidate time already included in the 
current commercial break is above a specified threshold 
value; 

determining whether the probability of the 
duration of the current commercial break, if the 
candidate t,ime being evaluated is added to the current 
commercial break, is above a specified threshold value; 
and 

determining whether each probability of the 
temporal separation between the candidate time being 
evaluated and an already existing commercial break, if 
any, is above a specified threshold value, wherein if 
each of the three probabilities is above the 
corresponding specified threshold value, the candidate 
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time being evaluated is added to the current commercial 
i 

break; 

determining whether there are candidate times having an 
adjusted score above a specified threshold value that have 
not yet been included in a commercial break and have not yet 
been excluded from inclusion in a commercial break; 

if there are candidate times having an adjusted score 
above the specified threshold value that have not yet been 
included in a commercial break and have not yet been 
excluded from inclusion in a commercial break, performing 
the steps of: 

selecting the candidate time having the highest 
adjusted score above the specified threshold value that 
has not yet been included in a commercial break and has 
not yet been excluded from inclusion in a commercial 
break; 

determining whether each probability of the 
temporal separation between the selected candidate time 
and an already existing commercial break is above a 
specified threshold value; 

if each probability is above the specified 
threshold value, performing the steps of: 

identifying the selected candidate time as a 

commercial start and/or end time in a current 

commercial break; and 

repeating the step of successively 

evaluating; and 
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if each probability is not above the specified 

threshold value, performing the steps of: 

excluding the selected candidate time from 

inclusion in a commercial break; and 

repeating the step of determining whether 

there ( are candidate times having an adjusted score 

above a specified threshold value that have not 

yet been included in a commercial break and have 

not yet been excluded from inclusion in a 

commercial break; and 

if there are no candidate times having an adjusted 

score above the specified threshold value that have not yet 

been included in a commercial break and have not yet been 

excluded from inclusion in a commercial break, performing 

the step of identifying the start and end time of each 

commercial break and the start time of each commercial in 

each commercial break. 

i 

34. (New) A method as in Claim 33, wherein the step of 
constructing further comprises the steps of: 

determining whether there are any candidate times 
having an adjusted score above a specified threshold value 
that have not yet been included in a commercial break; 

if there are candidate times having an adjusted score 
above the specified threshold value that have not yet been 
included in a commercial break, performing the steps of: 
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for each such candidate time that has not yet been 
included in a commercial break, performing the steps 
of: 

identifying the most temporally proximate 
commercial break to the candidate time; 

determining whether the probability of the 
duration of the most temporally proximate 
commercial break, if the candidate time is added 
to that commercial break, is above a specified 
threshold value; and 

determining whether each probability of the 
temporal separation between the candidate time and 
a commercial break other than the most temporally 
proximate commercial break, if any, is above a 
specified threshold value, wherein if each of the 
two probabilities are above the corresponding 
specified threshold value, the candidate time is 
added to the most temporally proximate commercial 
break; 

if the candidate time is added to the most 
temporally proximate commercial break, performing 
the stteps of: 

repeating the step of successively 
evaluating, for each candidate time having an 
adjusted score above a specified threshold 
value that has not yet been included in a 
commercial break, wherein the resulting 

- 68 - 



commercial break is a modified commercial 
break; 

for each candidate time of the modified 

commercial break, performing the steps of: 

i 

computing the average of the 
probabilities of each temporal 
separation between the candidate time 
and a temporally adjacent candidate 
time; 

if the average is below a specified 
threshold value, performing the steps 
of: 

if the adjusted score of the 
candidate time is lower than the 
average adjusted score of the 

i 

adjacent candidate times, 
eliminating the candidate time; and 

if the adjusted score of the 
candidate time is not lower than 
the average adjusted score of the 
adjacent candidate times, 
eliminating any adjacent candidate 
time that is less than a specified 
duration of time from the candidate 
time; 



computing, the average score of all of 
the candidate times of the modified 
commercial break; 

comparing the average score of all of 
the candidate times of the modified 
commercial break to the average score of all 
of the candidate times of the commercial 
break before modification; 

i if the average score of all of the 
candidate times of the modified commercial 
break is greater than the average score of 
all of the candidate times of the original 
commercial break before modification, 
replacing the original commercial break with 
the modified commercial break; and 

if the average score of all of the 
candidate times of the modified commercial 
break is not greater than the average score 
of all of the candidate times of the original 
cpmmercial break before modification, 
retaining the original commercial break; and 
if the candidate time is not added to the 
most temporally proximate commercial break, 
performing the step of excluding the candidate 
time from inclusion in a commercial break; and 
determining whether a modified commercial break 
replaced an original commercial break; 
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if a modified commercial break replaced an 

original commercial break, repeating the step of 

i 

determining whether there are any candidate times 
having an adjusted score above a specified threshold 
value that have not yet been included in a commercial 
break and the conditional step appropriate for the 
result of that determination; and 

if no modified commercial break replaced an 
original commercial break, performing the step of 
maintaining the identification of the start and end 
time of each commercial break and the start time of 
each commercial in each commercial break as existent 
prior to the step of determining whether there are 
candidate times having an adjusted score above a 
specified threshold value that have not yet been 
included in a commercial break; and 
if there are no candidate times having an adjusted 
score above the specified threshold value that have not yet 
been included in a commercial break, performing the step of 
maintaining the identification of the start and end time of 
each commercial break and the start time of each commercial 
in each commercial break as existent prior to the step of 
determining whether there are candidate times having an 
adjusted score above a specified threshold value that have 
not yet been included in a commercial break. 
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35. (New) A method as in Claim 1, further comprising the 
step of editing the audiovisual content based on the detected 
commercial breaks . 

36. (New) A method as in Claim 35, wherein the step of 
editing the audiovisual content based on the detected commercial 
breaks further comprises the step of deleting the audiovisual 
content representing a commercial. 

37. (New) A method as in Claim 35, wherein the step of 
editing the audiovisual content based on the detected commercial 
breaks further comprises the step of modifying the audiovisual 
content representing a commercial. 

38. (New) A method for viewing audiovisual content in which 
commercial breaks have been detected as in Claim 1 , comprising 
the step of skipping a commercial during viewing of the 
audiovisual content. 

t 

39. (New) A method for reviewing audiovisual content in 
which commercial breaks have been detected as in Claim 1 , 
comprising the step of searching for a commercial within the 
audiovisual content. 

40. (New) A method as in Claim 1 , wherein the audiovisual 
content is represented by a television signal. 
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41 . (New) A method as in Claim 1 , wherein the audiovisual 
content is represented by computer-readable data. 

42. (New) A method as in Claim 41, wherein audiovisual 
content is represented by computer-readable data acquired via a 
computer network. 

43. (New) A method as in Claim 42, wherein audiovisual 
content is represented by computer-readable data acquired via the 
Internet . 

44. (New) A method as in Claim 1 , wherein the method 
detects the one or more commercial breaks in real time as the 
audiovisual content is acquired for display by a display device. 

45. (New) A method for detecting one or more commercial 
breaks in a set of audiovisual content spanning a duration of 
time, each commercial break including one or more commercials, 
the method comprising the steps of: 

selecting a plurality of times within the duration of 
time spanned by the set of audiovisual content as a current 
set of commercial starting and/or ending times; 

selecting a t revised set of commercial starting and/or 
ending times including the current set of commercial 
starting and/or ending times and one or more additional 
times within the duration of time spanned by the set of 
audiovisual content; and 
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comparing the revised set of commercial starting and/or 

ending times to the current set of commercial starting 

and/or ending times to determine whether the revised set of 

commercial starting and/or ending times constitute a better 

set of commercial starting and/or ending times than the 

i 

current set of commercial starting and/or ending times , 
wherein: 

if not, the method further comprises the step of 
identifying the current set of commercial starting 
and/or ending times as a final set of commercial 
starling and/or ending times; and 

if so, the method further comprises the steps of: 
identifying the revised set of commercial 

starting and/or ending times as the current set of 

commercial starting and/or ending times; 

performing the step of selecting a revised 

i 

set of commercial starting and/or ending times; 
and 

performing the step of comparing the revised 
set of commercial starting and/or ending times to 
the current set of commercial starting and/or 
ending times. 
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46. (New) A method as in Claim 45, wherein the step of 
selecting a current set of commercial starting and/or ending 
times further comprises the steps of: 

identifying candidate times within the duration of time 
spanned by the set of audiovisual content, each candidate 
time representing a possible starting and/or ending time of 
a commercial; and 

selecting candidate times as one of the current set of 
commercial starting and ending times based on an evaluation 
of each candidate time to determine a likelihood that the 
candidate time is a commercial starting time or ending time. 



47. (New) A method for detecting a commercial in a set of 
audiovisual content spanning a duration of time, the method 
comprising the steps of: 

identifying a candidate time within the duration of 
time spanned by the set of audiovisual content, the 
candidate time representing a possible starting and/or 
ending time of a commercial; and 

evaluating the candidate time to determine a likelihood 
that the candidate time is a commercial starting time and/or 
ending time, wherein the evaluation is based at least in 
part on one or more characteristics of audiovisual content 
occurring after the candidate time. 
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48. (New) A method as in Claim 47, wherein the evaluation 

is further based, in part, on one or more characteristics of 

audiovisual content occurring before the candidate time. 

i 

49. (New) A method as in Claim 47, wherein the evaluation 
is based on one or more characteristics of audiovisual content 
occurring throughout the entire duration of time. 

50. (New) A method for detecting a commercial in a set of 
audiovisual content spanning a duration of time, the method 
comprising the steps of: 

identifying a candidate time within the duration of 
time spanned by the set of audiovisual content, the 
candidate time representing a possible starting and/or 
ending time of a 1 commercial; and 

evaluating the candidate time to determine a likelihood 
that the candidate time is a commercial starting time and/or 
ending time, wherein the evaluation is based at least in 
part on a relationship of the candidate time to one or more 
other candidate times. 

51 . (New) A method as in Claim 50, wherein the evaluation 
is based on a relationship of the candidate time to all other 
candidate times, 

i 
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52. (New) A method for detecting a commercial in a set of 
audiovisual content spanning a duration of time, the method 
comprising the steps of: 

evaluating the audiovisual content to identify the 
presence of a cue regarding the absence of a usually present 
network icon; and 

identifying a candidate time within the duration of 

time spanned by the set of audiovisual content based on an 

i 

evaluation of the identified cue, the candidate time 
representing a possible starting and/or ending time of a 
commercial. 

53. (New) A method as in Claim 52, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding a sequence of black frames in the 
visual content, wherein a candidate time is identified based on 
an evaluation of one or more network icon and/or black frame 
cues . 

54. (New) A me t'hod as in Claim 52, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding an audio pause in the audio content, 
wherein a candidate time is identified based on an evaluation of 
one or more network icon and/or audio pause cues. 
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55. (New) A method as in Claim 52, further comprising the 
step of evaluating th,e audiovisual content to identify the 
presence of a cue regarding a scene cut or fade in the visual 
content, wherein a candidate time is identified based on an 
evaluation of one or more network icon and/or scene cut /fade 
cues. 

56. (New) A method as in Claim 52,. further comprising the 

step of evaluating the audiovisual content to identify the 

presence of a cue regarding the occurrence of specified 

closed-captioning formatting signals and/or the absence of 

closed-captioning, wherein a candidate time is identified based 

on an evaluation of one or more network icon and/or closed- 

i 

captioning cues. 

57. (New) A method for detecting a commercial in a set of 
audiovisual content spanning a duration of time, the method 
comprising the steps of: 

evaluating the audiovisual content to identify the 
presence of a cue regarding the presence of music in the 
audio content; and 

identifying a candidate time within the duration of 

time spanned by the set of audiovisual content based on an 

evaluation of the identified cue, the candidate time 

I- 

representing a possible starting and/or ending time of a 
commercial . 
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58. (New) A method as in Claim 57, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding a sequence of black frames in the 
visual content, wherein a candidate time is identified based on 
an evaluation of one or more music and/or black frame cues. 

59. (New) A method as in Claim 57, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding an audio pause in the audio content, 
wherein a candidate time is identified based on an evaluation of 
one or more music and/or audio pause cues. 

60. (New) A method as in Claim 57, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding a scene cut or fade in the visual 
content, wherein a candidate time is identified based on an 
evaluation of one or more music and/or scene cut/fade cues. 

» 

61. (New) A method as in Claim 57, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding the occurrence of specified 
closed-captioning formatting signals and/or the absence of 
closed-captioning, wherein a candidate time is identified based 
on an evaluation of one or more music and/or closed-captioning 
cues . 
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62. (New) A method as in Claim 57 , further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding the absence of a usually present 
network icon, wherein a candidate time is identified based on an 
evaluation of one or more music and/or network icon cues. 

63. (New) A method for detecting a commercial in a set of 
audiovisual content spanning a duration of time, the method 
comprising the steps of: 

evaluating the audiovisual content to identify the 
presence of a cue regarding the density of scene cuts or 
fades in the visual content; and 

identifying a candidate time within the duration of 
time spanned by the set of audiovisual content based on an 
evaluation of the identified cue, the candidate time 
representing a possible starting and/or ending time of a 
commercial . 

64. (New) A method as in Claim 63, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding a sequence of black frames in the 
visual content, wherein a candidate time is identified based on 
an evaluation of one or more scene cut/fade density and/or black 
frame cues. 
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65 . (Ne „) A method as in Claim 63, further comprising the 
step of evaluating the audiovisual content to identify the 
preS ence of a oue regards an audio pause in the audio content, 
wher ei„ a candidate time is identified based on an evaluation of 
one or .ore scene cut/fade density and/or audio pause cues. 

1 

66 . (New) A method as in Claim 63, further comprising the 
st ep of evaluating the audiovisual content to identify the 
presence of a cue regarding a scene cut or fade in the visual 
cont ent, wherein a candidate time is identified based on an 
equation of one or more scene cut/fade density and/or scene 
cut/fade cues. 

67. (New) A method as in Claim 63, further comprising the 
step of evaluating the audiovisual content to identify the 
presence of a cue regarding the occurrence of specified 
closed-captioning formatting signals and/or the absence of 
closed-captioning, wherein a candidate time is identified based 
on an evaluation of one or more scene cut/fade density and/or 
closed-captioning cues. 



68, 



(New) A method for detecting a commercial in a set of 
audiovisual content spanning a duration of time, the method 

comprising the steps of: 

evaluating the audiovisual content to identify the 
presence of a cue regarding speaker identity; and 



i 
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identifying a candidate time within the duration of 
time spanned by the set of audiovisual content based on an 
evaluation of the identified cue, the candidate time 
representing a possible starting and/or ending time of a 
commercial . 

In the abstract: 

The abstract has been amended as follows: 

The invention [identifies] enables identification of 

starting and ending times of commercial breaks, as well as 

i 

starting and ending times of commercials within those commercial 

breaks, [can] to be found in audiovisual content (e.g., a 

television broadcast) using a method having characteristics which 

overcome disadvantages of previous commercial detection 

approaches. The invention is implemented as a solution to a 

"batch optimization" problem in which commercial locations within 

a set of audiovisual content are detected as a group by choosing 

a set of commercial locations which optimizes a cost function 

which can include consideration of, for example, 1 ) one or more 

of many types of visual recording, audio recording and/or 

closed-captioning cues, 2) relative locations of commercials 

i 

within the audiovisual content, and/or 3) probability models 
based on statistics obtained regarding characteristics of typical 
commercial and commercial breaks (e.g., commercial and commercial 
break duration, separation times of commercials and commercial 
breaks, likelihood of the presence of a commercial at any given 
time in a set of audiovisual content). Optimization can be done 

- 82 - 



over the total set of commercial location decisions, rather than 
on a per-commercial basis. Additionally, the cost function can 
be iteratively evaluated, increasing the accuracy of commercial 
location decisions produced by the method. Additionally, many 
more types of cues and combinations of cues can be used in 
detection of commercials in accordance with the invention than 
have been used in other approaches to commercial detection. 



t 
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